GAO 


Results  in  Brief 


United  States 

General  Accounting  Office 

Washington,  D.C.  20548 

- 1 

Acea*slc«  For  / 

OTTS-  GSfcW  ef'  ■■■ 

DYIC  fiB  □ 

U;iesijao  un  *  s  i  Q 

JuBtif 

National  Security  and 

International  Affairs  Division 

B-259020 

November  22, 1994  i 

The  Honorable  William  S.  Cohen 

Ranking  Minority  Member 

By 

Distrlbutioi?./  ! 

Availabllltir  Cods  3  | 

Dlst 

Avail 

Spec 

ial 

Subcommittee  on  Oversight  of  Government  Management 
Committee  on  Governmental  Affairs 
United  States  Senate 


Dear  Senator  Cohen: 

The  National  Aeronautics  and  Space  Administration’s  (nasa)  goal  is  to 
launch,  assemble,  and  operate  an  earth-orbiting  microgravity^  and  Ufe 
sciences  research  laboratory — International  Space  Station  Alpha — starting 
in  1997.  You  requested  that  we  review  NASA’s  efforts  to  develop  a  robust 
life  and  microgravity  sciences  research  community  for  the  space  station. 
Specifically,  we  reviewed  (1)  what  nasa  is  doing  to  assess  the  required  size 
of  the  research  community  needed  for  the  space  station  and  to  ensure  that 
such  a  community  will  be  available,  (2)  how  nasa  will  ensure  that  the 
research  selected  for  the  space  station  wUl  be  the  best  possible,  and 
(3)  whether  a  recently  canceled  Shuttle  research  flight  adversely  affected 
NASA’s  efforts  to  develop  a  research  community  for  the  space  station. 

In  June  1994,  we  reported  to  you  on  the  impact  of  the  expanded  Russian 
role  on  space  station  funding  and  research.  We  stated  that  Russian 
participation  in  the  space  station  would  substantially  increase  overall 
station  research  resources.  However,  the  degree  to  which  the  U.S. 
research  community  will  benefit  from  these  increased  resources  has  yet  to 
be  determined.^ 


NASA  has  taken  an  initial  step  to  assess  the  size  of  the  space  station’s 
research  community,  but  it  is  not  intending  to  develop  this  community  by 
directly  soliciting  proposals  to  do  research  on  the  space  station.  Instead, 
NASA  is  focusing  on  developing  a  comprehensive  research  program  that 
emphasizes  more  ground-based  research  and  uses  space  flight  only  for 
research  efforts  that  require  a  microgravity  environment  in  space.  To 


^Microgravity  is  a  condition  of  free-fall  within  a  gravitational  field  in  which  the  weight  of  an  object  is 
significantly  reduced  compared  to  its  weight  at  rest  on  Earth.  When  orbiting  Earth,  a  spacecraft  is  in 
continuous  free-fall  and,  thus,  in  microgravity.  Microgravity  is  also  a  low  acceleration  environment 
where  the  acceleration  imparted  to  an  object  is  one-millionth  of  that  measured  at  the  earth’s  surface. 
Following  NASA  usage,  the  term  “microgravity  science”  in  this  report  refers  to  the  study  of  chemical 
and  physical  phenomena  in  a  low  acceleration  environment. 

^Space  Station:  Impact  of  the  Expanded  Russian  Role  on  Funding  and  Research  (GAO/NSIAD-94-220, 
June  21, 1994). 
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accomplish  this  program,  nasa  wants  to  greatly  increase  the  number  of 
ground-based  investigators.  This  science-oriented  approach  is  reasonable, 
but  funding  levels  could  jeopardize  it  unless  nasa  adjusts  its  funding 
priorities.  To  achieve  its  goal,  NASA  would  need  increased  funding  for  its 
life  and  microgravity  sciences  research  and  analysis  over  fiscal  years  1996 
to  1999 — the  formative  years  for  the  initial  development  of  space 
station-related  research.  However,  nasa  expects  such  funding  to  remain 
constant.  The  result  could  be  a  smaller  than  desired  number  of 
investigators  in  the  ground-based  research  program  from  which 
station-based  research  will  be  selected. 

The  existing  process  of  using  peer  review  panels  to  judge  the  scientific 
merit  and  microgravity-related  relevance  of  experiments  for  the  space 
station  will  continue.  The  peer  review  scores  for  285  hfe  and  microgravity 
sciences  research  proposals  show  that  NASA’s  funding  decisions  were 
generally  consistent  with  the  findings  of  the  peer  review  panels:  most 
top-rated  proposals  and  none  of  the  lowest  rated  ones  were  funded.  But 
peer  review  panels  and  NASA  sometimes  disagreed  on  the  scientific  merit 
and  relevance  of  these  proposals.  For  example,  4  of  the  15  U.S. 
experiments  selected  by  nasa  for  a  July  1994  Space  Shuttle  research 
flight — ^the  International  Microgravity  Laboratory — were  not  among  those 
rated  highest  in  the  peer  review  process. 

NASA’s  efforts  to  increase  the  size  of  its  hfe  and  microgravity  sciences 
research  community  are  not  hkely  to  be  adversely  affected  by  the 
February  1994  cancellation  of  the  third  Spacelab  Life  Sciences  flight 
(sls-3).  The  U.S.  principal  investigators  on  that  flight  stated  that,  although 
they  were  concerned  about  nasa’s  lack  of  communication  about  the  flight 
cancellation,  most  of  them  have  been  accommodated  on  other  space 
flights  and  they  generally  will  be  able  to  meet  their  experiment  objectives. 
AH  of  the  investigators  with  whom  we  spoke  also  plan  to  continue 
submitting  proposals  for  future  nasa  research  opportunities. 


Background 


As  part  of  its  mandate  to  guide  the  nation’s  civil  space  program,  NASA  is  to 
preserve  U.S.  preeminence  in  critical  aspects  of  space  science,  technology, 
and  applications.  The  goal  of  hfe  and  microgravity  sciences  is  to  study 
gravit3''-dependent  physical  phenomena  and  those  phenomena  obscured  by 
the  effects  of  gravity  in  biological,  chemical,  and  physical  systems. 
Research  is  conducted  in  biotechnology  (e.g.,  protein  crystal  growth), 
combustion  science,  fluid  physics,  hfe  and  biomedical  sciences,  and 
materials  science.  Life  science  research  in  space  biology  studies  the 
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effects  of  gravity  on  living  systems  by  using  acceleration  enviromnents 
across  the  “gravity  continuum” — micro,  earth-normal,  and  hypergravity.^ 

NASA’s  Office  of  Life  and  Microgravity  Sciences  and  Apphcations,  which 
was  formed  in  1993,  funds  this  type  of  research.  Between  fiscal  years  1989 
and  1994,  the  annual  budget  authority  for  fife  and  microgravity  research 
increased  by  114  percent,  from  $222.4  miUion  to  $476.3  miUion.^ 

Not  aU  aspects  of  fife  and  microgravity  sciences  research  require  a 
space-based  environment.  Short  diuration,  low  acceleration  environments 
can  be  created  in  drop  towers  (2  to  5  seconds  of  free  fall),®  aircraft  flying  a 
distinctively  curved  flight  path  (up  to  23  seconds  of  low  gravity),  and 
suborbital  rockets  (over  300  seconds).  Hypergravity  can  be  created  by  a 
centrifuge.® 

Space-based  research  is  principally  conducted  in  pressurized  and 
nonpressurized  facfiities  on  the  Space  Shuttle.  The  centerpiece  for  this 
research  is  a  23-foot  by  16-foot  pressurized  module — Spacelab — ^that  fits  in 
the  Space  Shuttle  payload  bay.  Spacelab  was  developed  by  the  European 
Space  Agency  and  contains  utilities,  computers,  work  areas,  and 
instrument  racks  for  experiments.  An  exterior  cutaway  view  of  Spacelab  is 
shown  in  figmre  1. 


^Other  life  science  research  areas  are  space  physiology  and  countermeasures,  space  radiation  health, 
space  human  factors  engineering,  and  advanced  life  support. 

‘^Budget  authority  is  the  authority  provided  by  law  to  enter  into  financial  obligations  that  will  result  in 
immediate  or  future  outlays  of  federal  government  funds.  The  basic  forms  are  appropriations, 
borrowing  authority,  and  contract  authority. 

'^For  example,  experiments  dropped  from  the  89-foot  tower  at  NASA’s  Lewis  Research  Center  are  in 
free  fall  for  2.2  seconds. 

^A  centrifuge  is  a  rotating  device  designed  to  provide  a  high  acceleration  environment.  For  example, 
the  58-foot  diameter  centrifuge  at  NASA’s  Ames  Research  Center,  California,  can  expose  humans  to 
centrifugal  forces  up  to  20  times  Earth-normal  gravity  (20g).  Another  Ames  centrifuge  can  expose 
incubated  cells  to  centrifugal  forces  up  to  6g. 
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Figure  1:  Spacelab  in  Space  Shuttle  Payload  Bay 


Source:  NASA, 


The  most  recent  Spacelab  flight  was  the  second  International  Microgravity 
Laboratory  (iml-2),  which  ended  a  15-day  mission  on  the  Space  Shuttle  on 
July  23,  1994.  iml-2  was  a  collaborative  effort  by  NASA;  the  European  Space 
Agency;  and  the  national  space  agencies  of  Canada,  France,  Germany,  and 
Japan.  According  to  nasa  officials,  iml-2  provided  a  preview  of  the  science 
operations  to  come  on  the  space  station. 
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The  IML  flights,  which  began  in  January  1992,  gave  the  U.S  scientific 
community  access  to  foreign-developed  flight  hardware  wlule  providing 
the  international  research  community  with  access  to  the  Space 
Shuttle/Spacelab.  Approximately  80  investigations  were  performed  on 
iml-2,  including  15  U.S.  experiments — 11  in  the  biotechnology,  fluid 
physics,  and  materials  science  and  4  in  the  life  sciences.^  iml-2  was  the  last 
flight  of  this  international  series  of  spacelabs  before  the  station  era  begins 
in  1997.®  An  interior  view  of  iml-2  is  shoAvn  in  figure  2. 


^The  number  of  U.S.  experiments  excludes  the  Space  Acceleration  Measurement  System  (SAMS)  and 
Extended  Duration  Orbiter  Medical  Program  (EDOMP).  SAMS  is  designed  to  measure  and  record 
low-gravity  accelerations  at  three  experiment  sites  simultaneously.  EDOMP  gathers  data  on  the  effects 
of  long  duration  exposure  to  microgravity  on  human  physiology. 

^Three  more  pressurized  Spacelabs  are  scheduled  to  fly  on  the  Shuttle  before  the  start  of  the  station 
era:  U.  S.  Microgravity  Laboratory-2,  1995;  a  recently  announced  Life  and  Microgravity  Spacelab,  1996; 
and  Microgravity  Science  Laboratory-1,  1997.  Another  Spacelab — Neurolab — is  scheduled  to  fly  in 
1998. 
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Figure  2:  IML-2 


Source:  NASA. 


NASA  publicly  solicits  research  proposals  from  investigators  in  the  hfe  and 
microgravity  research  communities.  The  funding  decision  is  principally 
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based  on  an  evaluation  of  the  project’s  scientific  merit  by  a  peer  review 
panel,  nasa’s  peer  review  and  other  quality  assurance  procedures  are 
outlined  in  appendix  1. 


Developing  a  Larger 
Research  Community 
Will  Require  Ac^justing 
Fimding  Priorities 


NASA  intends  to  build  a  space  station-era  research  community  from  the 
ground  up.  To  do  so,  a  larger  cadre  of  ground-based  researchers  than 
currently  available  will  be  needed  to  adequately  support  U.S.  research  on 
the  station.  A  NASA  official  estimates  that  the  number  of  ground-based 
microgravity  researchers  needs  to  increase  from  73  to  240  between  fiscal 
years  1992  and  1998.®  nasa  officials  have  not  made  comparable  estimates 
for  life  science  researchers.  To  accomplish  this  goal,  nasa  has  abandoned 
its  tradition — ^principally  associated  with  life  science  research — of 
soliciting  research  proposals  for  general  and  specific  space  flight 
opportunities.  Although  this  approach  appears  reasonable,  the  planned 
funding  levels  do  not  match  the  program’s  objective,  and  funding  priorities 
may  need  to  be  reassessed  If  the  number  of  life  and  microgravity 
ground-based  investigators  is  to  be  significantly  increased.^® 


In  recent  years,  nasa  has  used  two  approaches  for  developing  life  and 
microgravLty  science  research  communities — “select  for  flight”  or  “select 
for  science.”  In  the  select-for-flight  approach,  aU  of  the  U.S.  life  science 
and  most  of  the  U.S.  microgravity  investigators  on  iml-2  were  selected 
from  proposals  submitted  in  response  to  flight-related  announcements.  In 
the  select-for-science  approach,  two  iml-2  microgravity  investigators  were 
selected  from  researchers  who  submitted  proposals  in  response  to  two 
1991  discipline-related  (“fundamental  science”  and  “biotechnology”) 
research  annoimcements. 

A  NASA  program  scientist  considers  the  iml-2  flight  to  have  been  a 
programmatic  success  and,  in  some  respects,  a  model  for  the  international 
space  station.  According  to  a  nasa  official,  one  indication  of  the  flight’s 
success  was  the  amount  of  good  research  generated  from  the  many 
proposals  submitted  in  response  to  a  mix  of  science  and  flight-related 
research  announcements.  Additionally,  two  of  nasa’s  recent  research 
announcements  were  in  the  select-for-flight  tradition:  its  July  1993 


Selecting  IML-2 
Investigators  Illustrated 
Two  Basic  Approaches  for 
Developing  Research 
Community 


®The  estimate  assumes  a  requirement  for  two  flight  investigators  for  each  research  discipline  each 
year,  and  four  ground-based  investigators  for  each  flight  investigator. 

^®NASA  is  not  necessarily  the  only  source  of  federal  funds  for  station-era  research.  For  example,  NASA 
and  several  institutes  of  the  National  Institutes  of  Health  have  agreed  to  establish  collaborative  science 
planning  and  joint  funding  of  research. 
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announcement  soliciting  proposals  for  research  on  a  1998  space  life 
sciences  flight  (Neurolab)  and  its  February  1994  announcement  soliciting 
proposals  for  Mfe  science  research  on  the  Russian  space  station  Mir  from 
1995  to  1997.  Presumably  then,  one  effective  way  to  develop  a  research 
community  would  be  to  solicit  specific  proposals  for  research  that  are 
directly  related  to  the  space  station,  nasa,  however,  has  chosen  to  move 
toward  exclusive  use  of  “select  for  science,”  as  discussed  below. 


“Select  for  Science”  Has 
Become  Preferred 
Approach 


Although  NASA  recently  solicited  proposals  specifically  for  research  on 
Shuttle  flights  to  Mir,  NASA’s  life  science  office  changed  from  the 
widespread  use  of  the  select-for-flight  approach  in  December  1993.  At  that 
time,  it  solicited  proposals  for  ground-based  research  in  space  biology 
focused  on  the  hypergravity  effects  that  can  be  induced  by  NASA’s 
centrifuges,  nasa’s  shift  to  ground-based  research  did  not  stifle 
competition  for  funding:  it  received  650  responses  to  the  December  1993 
announcement.  Although  “select  for  science”  is  relatively  new  to  life 
science  research,  all  microgravity  research  announcements  since  1990 
have  focused  on  research  opportunities  in  one  or  more  science 
disciplines.  “  And,  as  if  to  emphasize  the  independence  of  microgravity 
research  from  space  station  development,  nasa  changed  the  fiscal  year 
1992  goals  for  the  microgravity  program.  The  previous  goals  referred  to 
developing  and  using  the  space  station,  whereas  the  current,  more  general 
goal  is  to  “enable  [microgravity]  research  ...  by  choosing  the  carrier  most 
appropriate  for  the  experiment.” 

Physical  events,  unlike  biological  processes,  can  be  meaningfully  observed 
under  the  short-duration  microgravity  conditions  afforded  by 
ground-based  facilities,  aircraft,  and  suborbital  rockets.  Consequently,  a 
ground-based  microgravity  research  investigator  does  not  always  have  to 
conduct  experiments  in  a  space  environment,  and  many  do  not.^^  For 
example,  of  the  51  principal  investigators  who  conducted  such  research  at 
NASA’s  Lewis  Research  Center  from  fiscal  years  1989  through  1993,  only  7 
have  been  principal  investigators  on  space-based  experiments,  including  a 
microgravity  Spacelab  flight  in  September  1995. 


June  1991  solicitation  for  research  in  gravitational  biology  was  in  the  select-for-science  tradition. 

^^An  advantage  of  a  ground-based  versus  space-based  research  program  is  that  ground-based 
researchers  can  more  readily  repeat  their  experiments  even  though  the  time  an  experiment  is  in  a  low 
acceleration  environment  is  comparatively  brief.  For  example,  at  NASA’s  Lewis  Research  Center,  over 
1,000  drops  per  year  were  made  in  the  2.2-second  drop  tower  in  1989  and  1990  before  operations  were 
reduced  because  the  facility  was  modernized. 
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Select-for-Science 
Approach  Reasonable 


In  June  1994,  nasa’s  life  and  microgravity  sciences  advisory  committee 
stated  that  it  supports  the  life  and  microgravity  sciences  programs  “in 
terms  of  their  scientific  contributions  independent  of  the  type  of  flight 
platform.”  And  the  advisory  committee  specifically  recommended  that 

“NASA  establish  a  vigorous  ground  based  research  program  focussing  on  gravitational 
biology  in  which  centrifuge  facilities  at  nasa  centers  are  utilized  for  exploring  science 
programs  aimed  at  forces  greater  than  Ig  [Earth-normal  gravity].” 


NASA’s  Strategy  for  using  the  select-for-science  approach  to  further  develop 
a  life  science  research  community  in  the  station-era  appears  reasonable 
based  on  the  experience  of  the  microgravity  sciences  community. 

First,  the  microgravity  sciences  research  community  has  been  growing. 
Principal  investigators  funded  for  microgravity  sciences  research 
increased  by  120  percent — from  89  in  calendar  year  1989  to  196  in  fiscal 
year  1993.  The  budget  authority  for  microgravity  sciences  increased  by 
130  percent,  from  $75.6  miUion  to  $173.9  mUhon  during  this  period.  The 
number  of  proposals  submitted  in  response  to  research  announcements 
also  generally  increased  during  this  period.  For  example,  although 
proposals  submitted  in  response  to  materials,  fluids,  and  fundamental 
(benchmark)  physics  research  announcements  decreased  from  397  in  1991 
to  217  in  1993,  those  responding  to 

•  combustion  physics  announcements  increased  from  65  in  1989  to  98  in 
1993, 

•  biotechnology  research  announcements  increased  from  94  in  1991  to  141 
in  1994,  and 

•  materials  and  fluids  research  announcements  increased  from  69  in  1990  to 
346  in  1991. 

Second,  the  microgravity  research  community  is  stable  but  not  stagnant. 
Fifty-five  percent  of  aU  microgravity  sciences  investigators  that  were 
funded  in  1989  were  also  funded  in  fiscal  year  1993.  This  core  group 
represents  25  percent  of  the  investigators  funded  in  fiscal  year  1993.  On 
the  other  hand,  44  percent  of  the  investigators  funded  in  1993  were  not 
funded  in  1992. 

Third,  nasa  is  attracting  new  investigators  to  its  microgravity  sciences 
program.  The  decline  in  proposals  (from  397  to  217)  submitted  in  response 
to  the  1993  materials,  fluids,  and  fundamental  physics  announcement  may 
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have  been  partly  caused  by  NASA’s  stated  piupose  of  encouraging  new 
investigators,  and  most  of  the  55  investigators  funded  from  this 
announcement  were  new  to  the  program.  Only  15  of  them  had  been 
previously  funded  by  NASA.  In  June  1994,  NASA’s  life  sciences  advisory 
subcommittee  recommended  that  NASA  use  this  approach  and  establish 
appropriate  categories  within  life  science  research  announcements  that 
recognize  and  encourage  new  investigators. 

Finally,  for  those  proposals  we  reviewed,  the  select-for-science  approach 
produced  relatively  fewer  low  peer  review  scores  than  the  select-for-flight 
approach. Figure  3  shows  that  8  percent  of  the  select-for-science 
proposals  received  peer  review  scores  in  the  bottom  category,  while 
32  percent  of  the  select-for-flight  proposals  received  scores  in  the  bottom 
category.*^ 


'^We  examined  peer  review  scores  of  290  proposals  submitted  in  response  to  4  NASA  research 
announcements  issued  in  1988,  1989,  and  1991.  The  U.S.  experiments  on  IML-2  were  selected  from 
these  proposals.  As  such,  they  are  not  a  random  sample  of  all  proposals  made  from  1988  to  1991  and 
are  not  necessarily  representative  of  proposals  made  in  other  years. 

'■'The  top,  middle,  and  bottom  score  categories  are  described  in  appendix  II. 
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Figure  3:  Research 
Proposals-Comparison  of  Selection 
Approaches 


Number  of  research  proposals 
66 


45%  47%  0% 
Science® 


26%  42%  32% 
Flight® 


Top*" 

Medium*’ 

Bottom*’ 


®Refer  to  selection  approach. 
^Categories  based  on  peer  review. 


Similarly,  as  figure  4  shows,  of  all  the  proposals  in  the  bottom  category, 
only  16  percent  were  in  the  select-for-science  tradition,  while  84  percent 
were  in  the  select-for-flight  tradition. 
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Figure  4:  Research 
Proposals-Comparison  of  Categories 
Based  on  Peer  Review  Scores 


75  Number  of  research  proposals 


[ 


J  Science^ 
]  Flight^ 


51 


16%  84% 

Bottom® 


^Categories  based  on  peer  review. 

^Refer  to  selection  approach. 

Many  of  the  proposals  submitted  in  response  to  NASA’s  research 
announcements  were  not  considered  scientifically  meritorious.  For 
example,  peer  review  panels  gave  129,  or  44  percent,  of  the  290  proposals 
we  reviewed  relatively  low  scientific  merit  scores. 


Expected  Funding  Will  Not  NASA’s  plans  to  expand  its  ground-based  research  program  are  not  realistic 
Support  NASA’s  Approach  based  on  planned  funding.  A  NASA  microgravity  research  official  estimates 

that  NASA  will  need  to  fund  about  240  ground-based  investigators  to 
support  a  station-based  microgravity  sciences  research  program.  In  fiscal 
year  1992,  nasa  funded  73  ground-based  investigators  in  microgravity 
sciences,  only  about  30  percent  of  the  future  need.  Ground-based  research 
is  funded  from  NASA’s  research  and  analysis  budget.  However,  nasa  does 
not  anticipate  that  this  budget  will  increase  for  fiscal  years  1995  through 
1999.  Annual  life  science  research  and  analysis  appropriations  are 
estimated  to  be  about  $51  mfilion  and  microgravity  sciences  at 
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Selecting  Quality 
Experiments 


$21.7  million.  To  deal  with  this  potential  mismatch  between  plans  and 
resources,  NASA’s  microgravity  sciences  office  has  proposed  that  the 
research  and  analysis  budget  be  augmented  by  research  and  development 
funds  used  to  support  nasa’s  space-based  research  program.  The  proposed 
amounts  are  $4.7  mUlion  for  fiscal  year  1996,  $12.2  million  for  1997,  and 
$22.2  million  for  1998.  According  to  NASA,  shifting  resources  in  this  way 
would  not  increase  its  overall  budget  authority. 

NASA’s  fife  and  microgravity  sciences  advisory  committee  concurred  with 
this  approach,  stating  in  June  1994  that  the  research  and  analysis  budgets 
are  the  “seed  which  provides  for  a  successful  flight  program”  and 
recommended  that  they  be  increased 


“where  they  are  proven  to  be  inadequate  to  support  the  intellectual  underpinning  of  the 
flight  program,  even  if  this  means  a  transfer  from  the  [research  and  development]  budget 
so  as  to  comply  with  overall  budget  constraints.” 


NASA’s  quality  assurance  procedures  start  with  a  series  of  external  and 
internal  reviews  designed  to  evaluate  the  merits  of  research  proposals. 
Peer  review  is  a  crucial  part  of  this  consensus-building  process.  The 
process  starts  with  individual  reviewers  independently  evaluating  each 
proposal  assigned  to  peer  review  panels.  The  reviewers  then  resolve  any 
differences  by  consensus  within  the  peer  review  panel.  The  panel’s  final 
determinations  are  not  binding  on  NASA’s  selection  officials,  and  NASA  can 
choose  proposals  other  than  those  highly  recommended  by  the  panel. 

In  June  1994,  we  reported  that  the  peer  review  processes  at  the  National 
Institutes  of  Health  (nih).  National  Science  Foundation,  and  National 
Endowment  for  the  Humanities  appear  to  be  working  well  and  that 
intrinsic  qualities  of  a  proposal  (e.g.,  research  design),  and  not 
characteristics  of  reviewers  or  applicants  (e.g.,  applicant’s  region, 
academic  rank,  or  employing  academic  department’s  prestige)  were 
important  factors  in  reviewers  scoring.^® 

In  1993,  the  Senate  Committee  on  Appropriations  directed  nasa  to  model 
its  peer  review  standards  after  nih.  Based  on  the  Committee’s  direction, 
NASA  requires  that 


^^Peer  Review:  Reforms  Needed  to  Ensure  Fairness  in  Federal  Agency  Grant  Selection 
(GAO/PEMD-94-1,  June  24,  1994). 
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•  all  research  proposals  be  reviewed  by  peers  for  scientific  merit  and 
relevance  (previously,  some  Ufe  science  research  conducted  by  NASA 
scientists  was  not  subject  to  peer  review); 

•  all  research  be  reviewed  by  peers  at  least  every  3  years; 

•  all  research  be  reviewed  for  progress  annually  and  for  the  performance  of 
its  objectives  at  least  every  3  years; 

»  peer  review  be  performed  by  the  best-qualified  individuals  available  in  the 
field  reviewed;  and 

•  peer  review  scores  provided  by  external  peer  review  groups  be  critical 
factors  determining  the  priority  for  initial  and  continued  funding  of 
research  projects  and  programs. 


Individual  Peer  Reviewers’ 
Scores  Are  Usually 
Consistent 


Peers  agreed  on  the  scientific  merit  of  73  percent  of  the  proposals  that  we 
reviewed,  including  all  but  1  of  the  15  U.S.  experiments  selected  for  iml-2. 
Table  1  shows  the  distribution  of  the  reviewers’  scores.  Peer  reviewers 
were  better  able  to  agree  on  proposals  having  top  scientific  merit  scores 
than  on  proposals  having  middle  or  bottom  scientific  merit  scores.  Peers 
gave  only  11  percent  (11  of  99)  of  the  top  proposals  dissimilar  scores;  in 
contrast,  they  gave  36  percent  (44  of  126)  of  proposals  dissimilar  scores  in 
the  middle  category  and  38  percent  (23  of  60)  scores  in  the  bottom 
category. 


The  logic  of  peer  review,  in  our  opinion,  rests,  in  part,  on  the  assumption 
that  two  or  more  peers  can  independently  agree  on  a  research 
experiment’s  scientific  merits.  For  example,  they  should  agree  on  the 
testability  of  the  proposed  hypothesis  and  the  relevance  and 
appropriateness  of  the  experimental  design.  As  such,  peers’  scores  for 
scientific  merit  of  any  given  proposal  ought  to  be  the  same  or  similar.*® 


‘•’Scientific  merit  was  defined  in  a  variety  of  ways  for  the  proposals  we  reviewed.  For  the  life  science 
proposals,  scientific  merit  included  the  experiment’s  hypothesis,  experimental  design,  significance, 
investigator’s  professional  experience,  and  adequacy  of  facilities.  For  the  microgravity  science 
proposals,  criteria  such  as  relevance  and  institutional  resources  were  separately  rated  from  scientific 
merit. 
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Table  1:  Distribution  of  Dissimilar  Peer  Review  Scores  for  Scientific  Merit _ 

NASA  research  announcements  total  number  of  proposals 
(Proposals  receiving  dissimilar  scores) 


Peer  review  score  category^ 

Life 

sciences 

1989 

Microgravity 
fundamental 
science  1991 

Microgravity 

biotechnology 

1991 

Microgravity 
flight  1988 

Total 

Top 

23(1) 

26(2) 

33  (4) 

17(4) 

99(11) 

Middle 

33  (7) 

15(6) 

47(17) 

31  (14) 

126  (44) 

Bottom 

11  (3) 

2(2) 

8(2) 

39(16) 

60  (23) 

Total 

67(11) 

43  (10) 

88  (23) 

87  (34) 

285  (78)'> 

^Top,  middle,  and  bottom  categories  and  similar/dissimilar  scores  are  described  in  appendix  II. 


^Investigators  submitted  31 9  proposals  in  response  to  these  announcements,  but  29  proposals 
were  considered  unresponsive  to  the  announcements’  objectives.  Some  of  the  unresponsive 
proposals  received  panel  scores  and  some  did  not.  To  be  consistent,  we  did  not  include  any  of 
them  in  this  table.  Individual  peer  review  scores  could  not  be  located  for  five  other  life 
science-related  proposals  and,  thus,  could  not  be  included  in  the  table. 


NASA  Selected  Proposals 
Despite  Peer  Review 
Concerns 


Table  2  shows  that  NASA’s  selecting  officials’  funding  decisions  were 
generally  congruent  with  the  findings  of  the  peer  review  panel.  Of  the 
84  proposals  funded,  73,  or  87  percent,  were  in  the  top  category  for 
scientific  merit  scores,  and  the  other  11  proposals  funded  were  in  the 
middle  category. 


Table  2:  Distribution  of  NASA  Funding  Decisions 


NASA  research  announcements  total  number  of  proposals 
(Proposals  funded) 

Peer  review  score  category 

Life 

sciences 

1989 

Microgravity 
fundamental 
science  1991 

Microgravity 

biotechnology 

1991 

Microgravity 
flight  1988 

Total 

Top 

25(18) 

26(17) 

33  (22) 

17(16) 

101(73) 

Middle 

35(2) 

15(0) 

47  (2) 

31  (7) 

128(11) 

Bottom 

12(0) 

2(0) 

8(0) 

39  (0) 

61  (0) 

Total 

72  (20) 

43  (17) 

88  (24) 

87  (23) 

290  (84) 

Determinations  of  peer  review  panels  are  not  binding  on  NASA’s  selection 
officials.  For  example,  nasa  selected  four  proposals  that  received  mid-level 
scores  by  the  peer  review  panel.  Based  on  an  average  of  peers’  individual 
scores,  three  of  them  would  have  been  in  the  top  category.  However,  in 


^^The  four  proposals  represent  36  percent  of  the  U.S.  microgravity  experiments  flown  on  IMLr2. 
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subsequent  deliberations,  the  peer  review  panel  members  placed  three 
proposals  in  the  middle  category  because  the 

•  need  for  the  microgravity  environment  of  space  was  not  compelling, 

•  experiment-related  issues  could  be  resolved  using  the  ground-based 
program,  and 

•  appropriateness  of  analytical  techniques  was  questionable. 

No  peer  review  panel  was  convened  for  the  fourth  proposal  because  the 
number  of  proposals  in  the  specific  area  of  investigation  was  too  small. 
Generally,  the  peer  reviewers  found  the  proposal  to  be  of  high  quality,  but 
they  also  noted  that  the  research  objectives,  although  compatible  with  the 
life  science  program,  were  inconsistent  with  the  microgravity  science 
program.  In  this  case,  the  investigator  did  not  propose  to  use  microgravity 
to  study  phenomena  whose  understanding  is  obscured  on  earth  by  the 
presence  of  gravity. 

After  the  peer  review  panel  completed  its  deliberations,  a  nasa 
categorization  committee  made  category  assignments  that  were  forwarded 
to  a  steering  committee.  The  categorization  committee  determined  that  the 
four  proposals  were,  in  the  words  of  a  NASA  official,  “sound  but  not 
exceptional  science” — the  second  highest  of  four  possible  categories.^® 

The  steering  committee  assessed  these  categorizations  and  recommended 
funding  the  proposals,  but  committee  members  noted  that  one 
investigation  resembled  a  “fishing  expedition,”  another  had  “similar 
weaknesses”  to  proposals  that  were  rejected,  a  third  would  require  too 
much  time  to  conduct  on  a  Spacelab  mission,  and  a  fourth  should  only  be 
partly  funded.  Despite  these  views,  these  four  proposals  were  funded  for 
iml-2. 


Assessing  the  Impact 
of  Canceled  Spacelab 
Flight  on  the  Research 
Community 


NASA’s  efforts  to  develop  a  research  community  are  not  likely  to  be 
adversely  affected  by  the  February  1994  cancellation  of  sls-*3.  The  U.S. 
principal  investigators  on  the  Spacelab  flight  stated  that  they  will  be  able 
to  meet  their  experiment  objectives  on  other  missions,  including  multiple 
Shuttle  flights  to  Mir,  They  plan  to  submit  proposals  for  future  nasa 
research  opportunities. 


^^Categorizations  are  formally  described  in  the  Federal  Acquisition  Regulation  (48  C.F.R.  1870. 103). 
Informally,  the  highest  category  means  “best  science,  with  no  technical  risk.”  Proposals  in  categories  I 
and  II  are  “recommended  for  acceptance”  according  to  this  part  of  the  regulation.  Categorizations 
apply  to  proposals  submitted  in  response  to  an  “Announcement  of  Opportunity"  to  participate  in 
specific  NASA  programs  such  as  Space  Shuttle/Spacelab  flights  (48  C.F.R.  1870.1),  and  not  to  a  “NASA 
Research  Announcement”  used  to  solicit  proposals  in  areas  that  NASA  has  special  research  interests 
(48  C.F.R.  1870.2). 
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NASA  Advisory  Committee 
Concerned  About 
Cancellation  of  Spacelab 
Program  and  Usefulness  of 
Russian  Space  Station 


This  agreement  raised  the  concern  that  multiple  Shuttle  flights  to  Mir 
would  displace  future  Spacelab  flights  to  the  detriment  of  nasa’s  life  and 
microgravity  sciences  program.  Eventually,  these  concerns  were  formally 
expressed  by  the  Space  Studies  Board  of  the  National  Research  Council  in 
a  February  25,  1994,  letter  to  the  NASA  Administrator.  The  Board  discussed 
broad  issues  of  science  selection  and  management  in  the  space  station 
program,  including  termination  of  Spacelab  missions  because  these 
laboratories  “can  provide  more  high-quality  science  than  can  Mir . . . .”  The 
Board  also  noted  that 

“cancellation  of  [a  future  life  science  Spacelab]  mission  or  substitution  of  [Shuttle] 
middeck  experiments  for  a  dedicated  Spacelab  mission  would  have  serious  consequences 
for . . .  the  continued  participation  of  the  mainstream  life  sciences  community  that  nasa 
seeks  to  attract.” 

NASA  responded  to  these  concerns  in  April  1994,  stating  that  aU 
experiments  from  sls-3  have  been  accommodated  on  other  missions, 
including  Shuttle  flights  to  Mir.  nasa  also  noted  that  although  Mir  is  not  a 
substitute  for  Spacelab,  it  will  augment  and  enhance  on-orbit  science 
capabilities  because  experiments  requiring  more  than  30  days  of 
microgravity  cannot  be  performed  on  Spacelab. 

In  late  1994,  nasa  announced  a  new  life  and  microgravity  sciences 
spacelab  mission  for  July  or  August  1996.  This  mission  wUl  provide  a  flight 
opportunity  for  some  experiments  that  were  scheduled  to  fly  on  sls-3. 


NASA  planned  to  fly  a  collaborative  U.S.-French  sls-3  mission  in 
February  1996.  The  purpose  of  the  mission  was  to  study  the  effects  of 
microgravity  on  the  musculoskeletal  system  of  humans.  Rhesus  monkeys, 
and  rats.  The  French  were  responsible  for  developing  the  Rhesus  research 
facility.  Planning  for  the  mission  began  in  the  late  1970s.  On  February  18, 
1994,  however,  the  nasa  Administrator  notified  his  French  counterpart  that 
the  flight  was  canceled  because  of  budget  limitations  and  nasa’s 
commitment  to  the  international  space  station.  In  November  1993,  as  part 
of  the  agreement  between  nasa  and  the  Russian  Space  Agency  to  bring 
Russia  into  the  space  station  program,  the  United  States  and  Russia  ^eed 
to  fly  up  to  10  Space  Shuttle  flights  to  the  Russian  space  station  Mir. 
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U.S.  Investigators  Not 
Adversely  Affected  by 
Spacelab  Cancellation 


We  discussed  the  cancellation  of  sls-3  with  13  of  the  15  U.S.  investigators 
who  were  scheduled  to  fly  experiments  on  this  flight.^®  Their  views  are 
summarized,  as  follows: 

•  Nine  investigators  were  generally  satisfied  with  the  way  nasa  handled  the 
cancellation.  NASA  never  formally  notified  investigators  about  its  decision 
to  cancel  the  flight.  Consequently,  most  investigators  learned  of  the 
cancellation  from  rumors  or  other  informal  communication.  One 
investigator  said  investigators  should  have  been  consulted  before  nasa 
canceled  the  mission.  One  investigator  questioned  why  sls-3,  a  relatively 
near-term  mission,  was  canceled  rather  than  a  later  one  such  as  the 
Neurolab  flight  in  1998. 

•  Eleven  investigators  said  that  their  experiments  will  be  accommodated  on 
other  missions,  including  the  Russian  BiosateUite,  another  Spacelab 
mission,  or  Space  Shuttle  flights  to  Mir.^®  Two  investigators  said  they  have 
not  been  assigned  to  specific  missions. 

•  Ten  investigators  currently  scheduled  for  other  missions  said  they  wiU  be 
able  to  meet  their  basic  experiment  objectives.  However,  three  of  them 
said  they  will  not  necessarily  be  able  to  obtain  the  same  amount  of 
information  as  they  would  have  on  sls-3.  Their  experiments  involved  the 
use  of  Rhesus  monkeys,  and  even  though  they  will  fly  on  the  BiosateUite, 
in-flight  biological  measurements  cannot  be  done.  Three  other 
investigators  said  that  their  experiments  on  a  substitute  mission  would  be 
adversely  affected  by  hardware  Umitations  or  the  loss  of  opportunities  to 
efficiently  coUaborate  with  other  investigators. 

•  AU  13  investigators  said  they  wUl  continue  to  submit  proposals  for  future 
NASA  research  opportunities,  and  at  least  6  have  already  done  so. 


Scope  and 
Methodology 


To  accomplish  our  objectives,  we  obtained  documents  from  and 
interviewed  officials  at  nasa  headquarters  in  Washington,  D.C.,  and  at 
NASA’s  Lewis,  Johnson,  and  MarshaU  field  centers  in  Cleveland,  Ohio; 
Houston,  Texas;  and  HuntsvUle,  Alabama,  respectively.  In  May  1994,  we 
attended  the  iml-2  mission  simulation  and  science  review  conference  and 
observed  crew  training  exercises  prior  to  launch. 

To  review  the  further  development  of  NASA’s  Ufe  and  microgravity  sciences 
research  community,  we  obtained  information  on  research 


^^Two  investigators  did  not  reply  to  our  inquiries. 

“®The  BiosateUite  is  part  of  the  Russian  Cosmos  series  of  satellites.  Collaboration  between  U.S.  and 
Russian  scientists  on  Biosatellite-related  experiments  dates  from  1971  and  included  eight  flights 
starting  in  1975.  U.S.  investigators  started  flying  primate  experiments  on  this  satellite  in  1983. 
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announcements  issued  between  1988  and  1994  and  on  the  principal 
investigators  who  conducted  ground-  and  space-based  experiments.  We 
examined  peer  review-related  information  on  319  proposals  submitted  in 
response  to  the  4  research  announcements  related  to  iml-2.  We  categorized 
the  scores  of  all  proposals  that  peer  review  panels  considered  responsive 
to  the  objectives  of  the  announcements,  as  shown  in  appendix  11. 

To  assess  the  possible  impact  of  the  cancellation  of  the  sls-3  Spacelab 
flight  on  the  further  development  of  NASA’s  research  community,  we 
interviewed  13  of  the  15  U.S.  principal  investigators  on  that  mission. 

We  performed  our  work  between  November  1993  and  September  1994  in 
accordance  with  generally  accepted  government  auditing  standards.  As 
requested,  we  did  not  obtain  agency  comments  on  this  report.  However, 
we  discussed  the  issues  in  this  report  with  NASA  officials  and  incorporated 
their  comments  where  appropriate. 


As  agreed  with  your  office,  unless  you  publicly  announce  its  contents 
earlier,  we  plan  no  further  distribution  of  this  report  until  15  days  from  its 
issue  date.  At  that  time,  we  wUl  send  copies  to  the  NASA  Administrator  and 
other  appropriate  congressional  committees.  Copies  wUl  also  be  made 
available  to  other  interested  parties  on  request. 

Please  contact  me  on  (202)  512-8412  if  you  or  your  staff  have  any 
questions  concerning  this  report.  Major  contributors  to  this  report  are 
listed  in  appendix  111. 

Sincerely  yours. 


Donna  M.  HeivOin 
Director,  Defense  Management 
and  NASA  Issues 


Page  19 


GAO/NSIAD-95-33  Space  Station 


Contents 


Letter 

1 

Appendix  I 

Experiment  Quality 
Assurance  Procedures 

22 

Appendix  II 

Peer  Review  Panel 
Scores  for  Selected 
Projects 

26 

Appendix  III 

Major  Contributors  to 
This  Report 

27 

Tables 

Table  1:  Distribution  of  Dissimilar  Peer  Review  Scores  for 
Scientific  Merit 

Table  2:  Distribution  of  NASA  Funding  Decisions 

Table  11. 1:  Peer  Review  Panel  Scores  by  Announcement  and 
Relative  Category. 

15 

15 

26 

FigUrGS  Figure  1:  Spacelab  in  Space  Shuttle  Payload  Bay  4 

Figure  2:  IML-2  6 


Figure  3:  Research  Proposals-Comparison  of  Selection  1 1 

Approaches 

Figure  4:  Research  Proposals-Comparison  of  Categories  Based  on  12 

Peer  Review  Scores 

Figure  1. 1:  NASA  Fhght  Experiment  Definition  and  Development  23 

Process 


Abbreviations 

EDOMP  Extended  Duration  Orbiter  Medical  Program 

IML  International  Microgravity  Laboratory 

NASA  National  Aeronautics  and  Space  Administration 

NIH  National  Institutes  of  Health 

SAMS  Space  Acceleration  Measurement  System 

SLS  Spacelab  Life  Sciences 


Page  20 


GAO/NSIAD-95-33  Space  Station 


Page  21 


GAO/NSIAD-95-33  Space  Station 


Appendix  I 


Experiment  Quality  Assurance  Procedures 


The  National  Aeronautics  and  Space  Administration’s  (NASA)  guiding 
principle  for  quality  assurance  is  periodic  review  over  the  hfetime  of  an 
experiment.  Figure  1. 1  depicts  the  major  science  and  engineering  review 
milestones.^ 


'The  figure  illustrates  a  generic  process,  but  it  most  closely  matches  the  one  used  for 
micro  gravity-related  experiments. 
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Figure  1.1:  NASA  Flight  Experiment  Definition  and  Development  Process 


;||  Research  proposal  assessment 


Research  reassessment 
Flight  development 
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Experiment  Quality  Assurance  Procedures 


The  steps  or  actions  involved  throughout  this  process  are  outhned  below. 


Preparing  Research 
Announcements  and 
Soliciting  Research 
Proposals 


•  NASA’s  Discipline  Working  Group(s)  evaluates  research  program’s 
strengths  and  weaknesses  and  makes  recommendations  to  the  program 
scientist,  who  defines  areas  of  investigation  for  forthcoming 
announcement.^ 

•  NASA  conducts  workshops  for  prospective  investigators  from  the  scientific 
community  to  develop  interest  in  forthcoming  announcement. 

•  NASA  solicits  research  proposals  by  issuing  announcement. 


Assessing  Research  •  Peers  are  selected  by  contractor^  (Ufe  sciences)  or  NASA  (microgravity 

Proposals  sciences)  to  evaluate  proposals’  scientific  merit.  Peers  should  be  leading 

researchers  in  their  field  and  free  from  conflicts  of  interest  (e.g.,  a  current 
or  recent  professional  collaboration  with  an  apphcant),  and  not  currently 
receiving  research  funds  from  NASA. 

•  For  proposals  receiving  strong  science  reviews,  the  appropriate  NASA  field 
center  assesses  a  proposal’s  estimated  cost  and  engineering  feasibility.  For 
example,  Lewis  Research  Center  is  a  “center  of  exceUence”  for  two 
microgravity  science  disciphnes:  combustion  science  and  fluid  physics. 

•  NASA  program  scientist  recommends  principal  investigators’^  proposals  for 
funding  to  senior  nasa  management. 


Reassessing  Experiments 

Science  Concept  Review  •  Principal  investigator  and  project  scientist®  describe  science  scope  and 

feasibility  for  evaluation  by  Science  Review  Board. 

•  Project  manager^'  describes  conceptual  design  of  experiment-related 
hardware  for  evaluation  by  Engineering  Panel. 


^The  program  scientist  is  also  responsible  for  defining  a  mission’s  science  and  application  objectives 
and  ensuring  that  these  objectives  are  met. 

^Currently,  the  American  Institute  for  Biological  Science  convenes  peer  review  panels. 

■*The  principal  investigator  conceives  an  investigation  and  is  responsible  for  carrying  it  out  and 
reporting  the  results. 

^The  project  scientist  is  responsible  for  ensuring  that  (1)  the  principal  investigator  adequately  defines 
investigation’s  science  requirements  and  (2)  needed  experiment-related  hardware  will  accommodate 
required  science. 

^^The  project  manager  is  responsible  for  the  design,  development,  fabrication,  and  test  of  an 
experiment. 
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Experiment  Quality  Assurance  Procedures 


Requirements  Definition 
Review 


•  Project  manner  describes  cost  and  schedule  estimates. 

•  Engineering  panel  assesses  design  of  hardware. 

•  Science  panel  assesses  compatibility  of  science  requirements  with  design 
of  hardware. 


Developing  Flight 
Experiments 


•  Preliminary  design  review  assesses  the  compatibihty  of  science 
requirements  with  a  preliminary  engineering  model  (“breadboard”)  of 
hardware. 

•  Critical  design  review  assesses  complete  engineering  model  of  hardware. 

•  Preshipment  review  consists  of  experiment  simulations,  integration  with 
hardware,  and  testing  prior  to  sending  the  hardware  to  the  launch  site. 
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Peer  Review  Panel  Scores  for  Selected 
Projects 


To  determine  the  similarity/dissimilarity  of  peers’  perception  of  a 
proposal’s  scientific  merit,  we  defined  similar  scores  on  the  five-point 
scales  as  same  or  adjacent  scores  (for  example:  “3”  and  “3”,  or  “3”  and  “4”); 
and  on  the  nine-point  scales  as  same,  adjacent,  and  next  scores  (for 
example:  a  “3”  and  “3”,  or  "3”  and  “4”;  or  “3”,  “4”,  and  “6”). 


Table  11.1:  Peer  Review  Panel  Scores 
by  Announcement  and  Relative 
Category. 


Range  of  peer  review  panel  scores 

Announcement 

Top 

Middle 

Bottom 

Life  sciences,  1989^ 

1.0 -2.0 

2.1  -3.9 

4.0 -5.0 

Microgravity  fundannental  science,  1991^ 

9.0 -7.0 

6.9 -4.0 

3.9-  1.0 

Microgravity  biotechnology,  1991^^ 

CD 

b 

1 

b 

6.9 -4.0 

3.9-  1.0 

Microgravity  flight,  1988^ 

5.0  -4.0 

3.9 -2.1 

2.0-  1.0 

^Proposals  receiving  scores  greater  than  2.5  were  not  further  considered  by  NASA. 
•^Proposals  receiving  scores  less  than  7.0  were  not  further  considered  by  NASA. 
‘'Proposals  receiving  scores  less  than  3.0  were  not  further  considered  by  NASA. 
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